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ABSTRACT 



Citizenship 
the ICE project 



Evaluation results from the Improving 
Education (ICE) Project are presented. The purpose of 
was to design, and test a model for improving the 
political/citjizenship knowledge and attitudes of K-12 students by 
infusing citizenship education into an existing social studies 
curriculum. This evaluation examined the effectiveness of each of the 
five teacher support components of the change model (staff 
development, administrative support, cognitive and affective 
objectives, ^community resources, and materials) from the teacher's 
point of view (process evaluation). Also assessed was the impact of 
the curriculum change on student performance (product evaluation). 
Subjects 1 consisted of 17 project and control elementary and secondary 
teachers, 345 secondary students^ and 214 elementary students. The 
measurement instruments for the process evaluation included a 
multiple-choice instrument and questionnaires. Project and control 
students were pre- and posttested' using the Citizenship Knowledge 
Test and the Opinionnaire on Political Institutions and i 
Participation. Teacher evaluation results indicated that they made a 
significant gain in knowledge if rom staff development, objectives were 
very helpful, principals were J somewhat supportive, and the materials 
were very helpful. Product evaluation indicated that the ICE Project 
did significantly and practically affect the political knowledge and 
attitudes of secondary students; however, elementary students were 
not significantly affected. A re-evaluation of this project indicated 
that thW ICE project model does significantly affect the knowledge 
and attitudes of elementary students. Discussion is included on 
conclusions, limitations, and implications. (NE) 
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It is an understatement to describe the literature on social studies project * 
evaluation as sparse^ Social studies journals and books have provided 1 i ttlejjrf ormar 
tion on systematic evaluations of social studies curriculum projects. Two systematic 
reviews in the. recent literature (Wiley, 1977; Shaver, Davi^ and Helburn, 1979)' 
painted/broad murals of the status of the social studies, and although informative; 
they^contrilbujed only summaries of or brief commentary on evaluation research on* 
innovative social studies curriculum projects. Moreover, the. reviews were based pn 
evaluations which examined' "new social studies" p^jects of the sixties and seventies. 
Noc only were the projects dated, but evaluations often were based on impressionistic 
data or weakly designed studies. 

- The Elementary and Secondary Education Act (ESEA) of 197T ushered in a new era of 
educational improvement projects in the social studies and other subject areas. 
The focus of the development effort .Was shifted from research and development centers 
of universities and educational coroorations, whgre materials of the "new social 
studies" /had been designed to local school sites where curriculum improvement 
plans were to be designed and tested. Thus, the focus of evaluation shifted from 
Vesearch into the content and style of newly developed materials to outcomes in the 
classroom and processes in the school. 

The social studies literature lacks evaluation 1 reports or analyses orf ESEA, 
improvement projects. Some idea of the impact of the ESEA on the social studies 
can be gained from, more general educational literature '(Rand Corporation, 1978; 
Berry, -1979) a But here too the synthesis is not comprehensive. The. Rand studv 
, was based on a very small selection of local school projects. Also, it reoorted 
' projects established under federal ESEA guidel i^nes ^which preceded important amend- 
ments passed in 1977 and 1978. These amendments modified extensively the procedures 
for setting project objectives and designing projects to meet local needs. However, 
both the general literature and the social studies J iterature lack in deptb studies 



A paper presented to the Special interest Group/Research in Social Studies 
Education at the Annual Meeting of the American Education Research Association, 
Jtew;1ork, 1982. 



of the social studies curriculum improvement projects developed under^the most recent 

ESEA local 'development guidelines, . V 

The purpose of this paper is to, report the evaluation results from the 

^proving Citizenship Education (ICE) Proj^ct^which was initiated in 1977 under. 

existing ESEA guidelines and modified to meet revised 1978 state guidelines for 

* * * . - * « 

validation review. Besides documenting the effects of the ICE project on teachers v 

and students, the paper examines the potentials of local, social studies curriculum, 
projects which employ the same type of curriculum change model. 

' Improving Citizenship Education Model 

The ICE project (Fulton County, *Georgid)~ tested a change model which had five 
teacher support components. The five areas of support were: 1) .staff development - 
a teacher education program dealing with both the political science content and 
teaching strategies and materials; 2) administrator support - a series of meetings 

with principals, of the project schools informing them and°involving them in the 

* * 

project goals and methods; 3)' specific learning objectives - cognitive and affective 

fl * * i • 

objectives defined to guide the planning arid teaching of the project teachers; 

• * •» 

4) community resources - persons, agencies, organizations, and sites in the local 
area were fully identified by topic area to increase the pot^it'ial for first-hand 
learning experiences byvteachers and students (also a group, of community leaders 
served as an advisory board to the project staff);^5) materials - needed instruc- 
tional materials were identified and provided fpr teachers- (most needs were met with 
existing educational media, but some had to be developed by the staff). 

The expressed purpose of the ICE Project was to desi&j and test a model for 
improving the'political/citizenship knowledge and attitudes of 0 elementary/ and • * 
secondary students by infusing citizenship educatitnr into the existing social studies 
curriculum/ Nine educational goals (Table 1) guided the project's initial effort. 
From, these ni-ne goals > eight cognitive content areas were defined by a teacher committee 
and project staff (Table 2). From the goals and content areas, the staff and evalua- 
tion team derived aTset of cognitive and affective objectives! to aid ^evaluation of the 
Project (Table >3). Further, the relationship of specific objectives to goals was 
defined (Table 4). , ^ * 

Evaluatiorf Questions 

Original lyrtfre^eva-luation-pl-an for the project callecLfora. ten-weeks impl emen t 
tation; however, the demanf for more immediate ihformatiQn by the state department 
required a change to a six-weeks implementation in thes winter school quarter follow- 
ing the development of the teacher support components. The evaluation had two purooses 



I • 

: ' - " . Table 1 

Goals of the Improving Citizenship Education Project 1 

' , ■ 1 1 ■ : 

1. Understanding arid acceptance of the responsibilities of U.S. citizenship; 
\ 2. Understanding of the structure and operations of local, state, and national governments; 

3. Understanding of the roles of individuals ia the political decision making processes and developing the skills for 
participating. • - ■ 

4. Understanding of the principles of individual rights and learning to make choices in the context of concern for the 
society as a whole; _ 

5. Understanding of lheU.S. legal system and a' commitment to rule of law; . J 

6. Knowledge of current public issues and skills for evaluating alternative choices in regard to these ^sues; - / . : 

7. Understanding of the interdependence of th§ global community and the political processes in operation internationally; 

8. Understanding of the means to participate in school, local, state, and national political processes; 

9. Understanding of the need for government and community services and the ways to secure, utilize and contribute to 
<.' these services. 



Source: Margolis (1981) 
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* 1 - Table 2 

. - • ; . 

Content Areas of the Improving Citizenship Education Project 



1. Nationaf GovernmenVfederalism, bicameral lawmaking, executive powfer and foles, judicial powers, separation of 
powers, checks and balances, policy making processes, interest groups, lobbies, electoral college, two-party system, 

* bureaucracy, national revenues and expenditures/U.§. citizenship. ~ • 

• 2. State and Local Government-intergovernmental relationships, state and local lawmaking, state and local citizenship, 

services of state and local governments, financing s ervices , executive leadership, bureaucracy, local interest groups. 

3. Democratic Principles-popular sovereignty, limited powers of government, majority rule, minority rights, reserved 
. -powers, gene ri4« welfare, constitutional republic. , 

4. Politics-political symbols, campaigns, elections, voting behavior, representation, public opinion, means. and limits of 
political power, political socialization, political leadership. 

5. Laiv and Individual Rights-rule of law, equality before the few, justice, the Bill of Rights, judicial principles, individual 
rights and responsibilities, limits of individual freedom. « - / 

6. Global/International Studies-interdependence, multi-national organizations, United Nations, treaties, balance of 
• power, international interest groups, traije agreements, deterrence, detente, international, assistance and foreign aid. 

7. Analytical Skills-making informed decisions on public issues, value analysis, evaluating informatio i and information 
sources, interpreting and evaluating quantitative data, distinguishing fact from opinion, synthesizinj [ data from charts 
and graphs. \ m * - 

8. Participation Skills-conflict resolution skills, news awareness skills, registration and voting skills, skills for jury duty 
and participation in judicial process, party and interest group participation, career awareness, securing basic govern- 
mental services. % I 



Source: Margolis (1981) 
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- Table 3 * ~ ' - 

•i - -i • * 

Specific Objectives foir the Improving Citizenship Education Project 

i • • '• ' ' 

1.0 Know and apply specific facts, basic concepts, and processes related to government and politics. 
LI. Identify specific facts, processes, and basic concepts of national government. 
| 1.2 Identify specific facts, processes, and basic concepts of state and local government. 

1.3 Identify specific facts, processes, and basic concepts of democracy. . 

1.4 Identify specific fccts, processes, and basic concepts of politics. 

1.5 Identify specific facts, processes, and basic concepts of law and Individual right*. 

1.6 Identify specific facts, processes, and basic concepts of global affairs. - 
1:7 Identify participation skills related to government and politics* 

"1.8 Utilize analytical skills with government and political data and issues. 

2.0 Demonstrate commitment to democratic institutions/ principles, and processes. 

2.1 Express, commitment to democratic Institutions. 

i* 2.11 Identifies magnitude of agreement with statements about political institutions as others should view them. 

2.12 Identifies magnitude of agreement with statements about<political institutions as self views them. 

1 2.2 Express commitment to community democratic processes. 

2.2f Identifies magnitude of agreement with statements about community democratic participation as others 
should view them. 

2.22 Identifies magnitude of agreement with statements about community participation as self view*. them. 

. .2.3 Express commitment to school democratic processes. * ' 

» * «> 

2.31 Identifies magnitude of agreement with statements about school democratic participation as others should 
* view them. 

2.32 Identifies magnitude of agreement with statements about school democratic participation as self views them. 
Source: Margolis (1981) , \ 
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Relationship between Goals arid Specific Objectives of 
the Improving .Citizenship Education Project 
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- ' >__^ i \ • . * - ■ ' 

1) to evaluate the effectiveness of' each of the five teacher support components of 
the»channe model - from the teachers--' point of view (process evaluation), and 2) to 
assess the.impactvof. the curriculum change^model on student performance (prpduct 
evaluation) . * \ . ' . \ ' ' ' ' 



Specific process evaluation questions were-: 



!• How much dick the teachers learn about teaching procedures in the staff 
s development session^? * • 



2. How supportivev did the teachers feel their administrators were during 

i mplementati on W the citizenship project?* ■ * <■> • ^ * 

*3. How useful did teachers'. £ind the delineated objectives of the program? 
4. How 'helpful did -the teachers find the cbmmunfty resounds identified^ 
\ for the project?' t \ . \ * 

ft How helpful did the teachers find the c^rriculuifi materials provided or 

V developed for the project?* 
» Specific product^evaluati on questions were: x 

1.. ^Was^ ft ere a significant and practical difference between the political 

knowledge of project and c contrQl^ students? 
2. Was* there a significant and practical difference between the political 
.attitudes of project" and control students? 
* The linking rationale (Rutman, 1977) .generated by project staff and. evaluators 
clarified the connection between processes and products and, was stated thus: 
< l Teachers will take parr in a staff development projVct which will make 
them aware of the specific objectives , cognitive contents areas, and goal s, of 
the project. In addition, teachers will be instructed in\the use of various 
activities* and teaching strategies to achieve the specific^ objectives, and 
informed of a variety of yseful materials (i.e., community Resources and 
tlocated or developed materials ). Teachers will be given administrative support 
by principals familiar wfth^the goals and objectives of the Wtfject.- With 
this background and. support, teachers will spend more time tfechfng.the content* 
4 topics* employ a greater number of activities and strategies, *ind use.a greater 
number of materials in their classrooms to achieve the specific object* 1 '^. * 
Therefore, students will be exposed to an improved citizenship education program 



and as a result students will increase ^heir achievement o^f the specific 

/ 



— objectj«e.s_of„thej)roject. \ _^ • ^ / 



It should be noted that two related elements yere evaluated* Orte, the 

curriculum change- model , a -process, was evaluated by operational izitvg citizenship * 

education as defined by 3 the specific objectives of the Project 1 *- Two, the effective- 

• * * . 

ness of "citizenship education 11 with students* as defined by the same objectives 

Was simultaneously examined. The data collected allowed for the evaluation of both 

elements. * 

Additionally, it'should be clarified that because of differing requirements 

for the teacher education portion of the pr?ofeesS and the differing 'instrumentation needed 

to measure product, elementary. amd secondary programs were evaluated separately 

\ * ** * 
using the same evaluation design, • 

Procedures for-£-va-luat-ion~ v _ 
Subjects * b ■> < 

• Teachers . Originally, there we're 12 secondary and 7 elementary teachers de- - 
signated for the formal evaluation of the ICE project. One teacher failed to 
complete the teacher education program and, another left the school system. There- 
fore, the final sample of project teachers consisted of 10 secondary^ eiijhth; 
6 ninth; and 1 ten-twelfth grade) and 7 elefnentary (1 first; 1 second; 1 fourth; 
2 fifth; and 2 sixth grade) teachers- , 

These teacners were from five secondary schools dnd s five elementary schools 
designated as, "project schools." 0 AVI had volunteered to participate in the 
improvement project. Based on information supplied by the school records pffice, 
the projecjt schools were representative of the 'diverse sqpio-economic characters- 
^ics in the school district and included ur&an, suburban,, and small town eommuni- - N 
ties and school populations ranging from nearly all white to. nearly all black 
students. Project schools were considered representative of the system's approxi- 
mately 35,000 students attending 60 elementary schools and 19 high schools. 1 

Volunteer teachers from schools matched with project schools served as control 
teachers. The ten secondary and seven elementary control teachers were further 
matched Y by grade level, and in the secondary schools by, subject taught. 

students. The students involved in the product evaluation were in classes 
taught by teacher subjects. For the twenty secondary teachers, one class, in their 
schedules was randomly selectee^. Initially, there were 419 secondary (201 project; 
218 control) students and 262 elementary (134 project^ 128 control) students who f 
completed the pretests. However, the final sample size used in the evaluation > 
involved 345 secondary (165 nroject; 180 control) students and 214 elementary 



(llO projedt; 104 control) students*. The loss, of subjects- from protest' to fittest * 
representedWp.rcxiinately 18% of *alT .groupV The si2e of the loss reflected the - " - 
fact that only students -who took both pretests and posttests were included .in \he 
"final ^amp>e,\and/f\u accounted for a high percent of the absences./ It was 1 
administratively i'mpossi We to. retrieve the^missing data. . *. 

# . In the fiflal sample, -a total of 157 males (63 project; and 94cqntro.l) and 188 ' # 
females (102 'project; 86 control) were in the secondary student sample; and a total 
of 111 males (55\ project; 56 control) and 103 females (55 project; 48 control) were 
in the elementary student sample. The racial composition of the sa)npl* was 208'whfte - 
(106 project; 102* control) arid»13£ black (58 project; 79 control) students in the secon 
aary. sample; a v nd Tl7 white (54 project; ;63 control) ancf97 black«(C6 project; 41 
control) in the elementary student sample." * "~ 

Measurement .Instruments 

' Process instruments . To measure -the effect of the s£aff development component 
of the project^process, a 48item muVtiple choice instrument was constructed" to 
measure specific objectives of the staff development. Eight areas were covered on. 
the measure: Case Studies-, Community 'Studies, Values. Education,' Moral Education, 
News media analysis, Simulation, Quantitative Analysis, and Decision-Making. -The N 
instrument was scored right/wrong.. Scores could range f rom 'zero to 48 (with 6 
items per area).- Instrument validity was based 'on the dTrect relationship of the 
content of the items to the objectives of^'the staff developments Reli ability of the 
instrument was estimated using Kuder-Richardson Formula 21. ' Reliability for pretest 
scores was .66, and .=33 for posttest scores. / 

Additional^ data ^for the process evaluation was obtained from three questionnaires-.- 
Questionnaire on Management Support, Questionnaire on ICE Objectives, and Questionnaire 
on Materials including community resources (copies are found in Margolijs, 1981). . * 
Eaduquestionnaire had'two parts.' The first part asked teachers y td rate the support'ive- 
ness or helpfulness of the specific process component on a scale of 1( low) to 
4 (high). The second part asked teachers to review in more^detail the effectiveness 
of each support area. This second part was designed for obtaining descriptive 
information for the project staff. No 'formal -attempt was made to determine validity 
or. reliability of these questionnaires. , Informally, the instruments were verified 
by conversations with project teachers. . , • " . • 

Product instruments . The Citizenship Knowledge Test (CKT) was used to collect 
data on the first product evaluation questio'n.- This test has both secondary and 
elementary editions. The secondary. level test has two forms,. Form A, and Form B; 



and tfoe. elementary level test has two forms, Gractes 1-2, and Grades 3-7* 

• "Tfie secondary .level CKT has 73 items* in eight subtests related to the specific 
objectives listed, in Table, 3. / These are:-* National Government (11 items), State 
and Local Government; (10 items), Politics and Political Ethics (11 items), "Democratic 
Principles -(7 items), Law and individual Rights (10' items), Global Affairs^ (5 items) t 
'Participation Skills (10 tf&ns), and Analyst; Skills (9 items). Variation in number 

of items per subtest (objective) reflected the emphasis each subtest area was given 

* 

on the secondary level of the ICE project. The*tojtal test score for the secondary 
level CKT.could.jrange from zero to 73. * - 5 

Inr tfie Project evaluation, Form B of the secondary level CKT was used (for " * 
details on the .rati or\v\e for choosing Form B, see Hepburn and Strickland, J 979). 
The validity arid reliability <£f Form B have been ^documented elsewhere (Hepburn 
and Strickland, 1979; Jtepier, Hepburn and Strickland, 1981). Cronbach alpha procedure 
were used to calculate* reliability estimates of the total secondary student CKT scores 
which were .89' for the^pretest scores and .90 for the pqsttest, scores. . . 

On both* forms pf the elementary level -CKT there are 41 sterns in seven subtests, 
related to the specific objectives 1,^2, 3* 5, 6, 7, and 8 listed in Tabled. 
(The more abstract content of objective, 4 was not emphasized in the elementary grades) 
The seven .suotests are: National Government (6 'items K State and local Gpvernment 
(8 items), Politics and.Political .Ethics (7 items), Law and 'individual Rights.- , ^ 
(7 items),. Global Affairs (5 items), Participation Skills (4 items), and Analytic'' 
Skills (4,*items). Again,- ths variation in the.number of items for each subtest 
. rfefTected'the emphasis each subtest area was given on the elementary level 'of*th* 
ICE project-. The*major difference between the Grade 1-2 and Grades 3-7 forms' of the 
elementary CKT is tn the complexity and difficulty of the wording used m the item 

stem or distractors. Basically the two forms are parallel.* TotTal teit scores, could 

jt ' 

range from zgro to 41 . / " * . 

* ^ - * t 

Both forms of the elementary J evel CJ<T were used /in the evaluation. * The validity 

• ... 

and reliability of the elementary level CKT is documented elsewhere (Napier, \ 

Hepburn ,^and Strickland, 1981). Jn the. project evaluation, Gnonbach 'alpha relia* 

bflity estimates- for, pretest scores were .83, and for posttest scores -.88. ^ 

The Opiniorma'ire-on Political Institutions and Participation (0PIP)>as used' 

to, collect data on the second product evaluation question. This test also has both 

secondary and elementary editions which are parallel. The OPIR is a 48-ifcem instru- 

— — / 

ment w^'th six subtests each containing eight items. . 



The first and second subtests measure attitudes about political institutions 
(objectives 2.11 and 2.12). The first subtest, Political Institutions-Others, 

indicates the respondent's perception of how people in general sjiould view American 

" \ 

political institutions." The second subtest, Political Instltutions-^elf reflects 
the respondent's personal attitude toward American political institutions. 

The third and fourth subtests assess attitudes related to participation in public 
political processes (objectives 2.21 and 2^22). The third subtest, Public Political 
Parti ci pat ion-Others, reflects the respondent's perception of whether individuals 
should participate in public political processes. The fourth subtest, Public 
Political Participation-Self, measures the respondent's attitude toward personal 
political participation. / 

The fifth and sixth subtests measure attitudes towwd^ participation in school 
political processes (objectives^.. 31 and 2.32). The fifth subtest, School Political 
Participation-Others, reflects the respondent's perception of whether others should 
participate in school political processes. The sixth subtest, School Political 
Participation-Self, ^reflects, the respondent's attitude toward personal participation 
in school political processes. 

The secondary form is designed so that individuals respond to the 48 statements 
oh a 5-point scale (Strongly Agree, Agree, Uncertain, Disagree, or Strongly Disagree). 
On the elementary form, individuals respond to the statements on a 3-point scale 
(Agree, Uncertain, or Disagree). In this evaluation, the secondary form was scored 
on a 3-point scale to allow for comparison across elementary and secondary grades. 
Also, the instrument was. validated using that scale (Hepburn and Napisr, 1980a; Napier 
and Hepburn, 1981). Consequently, the overall score on both the secondary and 
elementary forms could range from 48 (very. negative) to*144 (very positive). 

The validity- and reliability of the secondary and elementary level OPIP are 
documented elsewhere (Hepburn and Napier, 1980a; Napier and Hepburn, 1981). Cronbach 
alpha* reliability estimates for secondary subjects .were .78 pretest and .89 posttest; 
for elementary subjects .84 pretest and .88 posttest. 

Treatment * * 

Teachers. Staff development for project teachers presented objectives of the 
ICE project and immersed them in teaching activities*, strategies, and materials for 
achieving the objectives. The group participated in a 45-hour staff development ^ 

_B.ro gram initiated in July, 1979, with a week of all-day sessions and followed up in 

- * - \ 

..the fall with five release-time days. An outline of the lessons and objectives for 
staff development "training is presented in the project's Implementation Handbook 
(Margolis, 1981). / 



* ♦ 

' * 12 . ' 

Administrators . * An all -day meeting was held with principals and project schools 
in 1978 to inform them of objectives and methods of the; project and enroll th6ir 

support. In 1979 the project "director met witfr principals individually to inform* 

t j » 

* l_„ them_qf Jhe_progi^ss ^ support. 

Students . Utilizing project objectives, materials, including drafc versions 

2 \ 
of the secondary or elementary handbooks ; and with administrative support tfye teach- 
ers conducted six-weeks units aimed at improving student political knowledge and 
attitudes. Each project teacher was free to operationalize the, units taught to fit 
the course and grade levels they were teaching,- T heref ore, some c heck was needed 
on whether they taught the content of the objectives,. used the fecommended activi- 
ites and strategies, and employed the suggested materials. To check on implementa- 
tion, a daily log was kept by each teacher (a copy, is in Margolis, 1981). For com- 
parison, control teachers also kept the same log. Analysis of the log data later 
indicated that secondary project teachers spent significanly more time teaching . 
the content related to the objectives and taught more citizenship topics. Also 
they used signficantly more recommended materials, and activities than the control 
teachers. However, the elementary project teachers did not spend significantly more 
time teaching the content, nor did they use significantly more topics, materials, or 
activities then the control teachers (see Table 5). Therefore, the effectiveness of 
'the elementary implementation was questionned (see discussion in results section). 

Data Collection * 

Process evaluation data . Project teachers took the 48-ifem multiple choice 
instrument as a pretest and again at the end of the 45 hour summer course. Project 
teachers responded to the questionnaires on objectives, management support, and 
materials at the end of the*six-week implementation in Winter 1980. Unfortunately, 
two of the seven elementary teachers failed to return the- three questionnaires despite 
repeated inquiries from the project siaff for the return of these measures. 

Product evaluation data . Project and control elementary and secondary students 
took the two pretests in early January 1980. The attitude test was given one day, 
and the knowledge test was given the next day. Both project and control subjects 
,took the posttests in March with the tests given in the same order. 

. Evaluation Re sults a 

Process Evaluation 

Secondary Teachers . Table 6 presents the descriptive data for s^conBary-^ 

teachers for staff development pretests and posttests and responses to the first part 
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Table 5 

Descriptive Statistics, F-Ratios, P-j Values for Number -of Materials, Activities, 
and Contents plus Time Covering Topics used by Secondary and Eleipentary 
~ Evaluation Project and Control Teaichersl 
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P 


Number of 
Materials 


Mean 
S.D 


7.6 
.0.8 


5.0 
1.8 


17.69 


.001 


5.3 

— 2.6. 


3.5 
3.2 .. 


1.25 


.288 


(Scale 1-8) 




















Number of 
Activities 
(Scale 1-8) 


Mean 
S.Dr 


7.0 
1.1 


4.6 

1.9 " 


a2.23 


.003 


5.6 
1.3 


4.2 
2.0 

— 


2.30 


.158 


Number"T>f 
Contents 
(Scale 1-8) 


Mean 
S.D. 


7.2 
0.8 


4.5 
2.9 


8.19 


.010 


6.4 
2.1 


4.3" 
2.0 


3.32 


.096 


Time on 
topics 


Mean 
S.D. 


1332.0 
234.1 


726.0 
529.8 ' 


10.95 


.004 


. 714/3. 
563.3 


280.0 , 
271.5 


2.95 


.114 



One Control teacher never returned the daily log 



. Table 6* 

r. S ' • 

Descriptive Statistics for Staff Development Pretest 
and Posttest plus Responses to Questionnaire on 
ICE Objectives, Management Support and Materials j 
for Evaluation Project" Teachers 



Source . 1 


Statistics 


. Secondary 

» 


Elementary 




Mean 


28.4 


23.0 


Pretest 










S.D 


5.8 


8.7. 


(Scale 0-4)3-) 


* 






N 


• 10 


7 




• Mean 


Si 

34. ~ - 


i ■ Zo . 9 


Posttest 










, S.D. 


3.7 


7.2 


(Scale 0-48) 








N • | 


10 

• 


7 


e 


Mean 


3.4 


3.6 • 


Objectives 








S.D. 


-0.7' 


0.6 


(Scale 1-4) ' * 








A 


N 


10. 




rianagemen u 


Ma on 

riean 

0 


0 A 
L . O 


Z ♦ 0 


Support 


S.D. 


l.l 


1.1 


(Scale 1-4) 


• 




* 


N 


10' 


5 




Mean . 


3.6 


3.6 


Materials 










S.D. 


0.5 


0.6 


(Scale 1-4) 






ju Q 




N 


10 


5 


•k 

Two Element. 


ft 

ary Teachers 


never returned 


questionnaires 
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of thethree questionnaires* A significant .gain in knowledge was made fronfthe staff 
development (t d - 3.34; df * 9; p <-.01). Questionnaire results indicated that the 
objectives were very helpful, principals were, somewhat supportive, and the materials 
were very helpful. * - 



Elementary teachers . Table 6 'also presents the same descriptive ^data for elemen- 
tary teachers. Like the secondary teacfters, elementary teachers mace a significant 
gain in knowledge from the staff development (t d - 5.03; df - 6; p<4 -01). Results^ 
from the questionnaires indicated that the elementary teachers found' the objectives 
very helpful, the principals were somewhat supportive, and the materials 'wer^e very 
helpful. i \ 

Product Evaluation 



For both the secondary and elementary data analysis of covariance procedures 
were used. The analysis was conducted blocking. for effects by sender and race: 
According to Cook and Campbell (1979), analysis of covariance blocking on variables 
related ta* % the dependent variable is a more powerful procedure than simple analysis 
of variance. The variables of .gender and race were used as blocking variables becaus* 
past studies have indicated there are relationships between these two variables 
'and political knowledge and attitudes (see Hepburn and Napier, 1980b). 

' Secondary, product. resul ts > 'Table 7 presents descriptive statistics for the 
secondary level students on the knowledge and attitude tests. Analysis of covariance 
on project and control results is presented in Table 8. A significant difference 
was found between project and control groups oh both the CKT and OPIP. 

The product evaluation questions asked about "practical " as well as statistical 
significance. Practical signfficance was determined by calculating "effect size" of 
statistically significant result. The effect size is a statistical "rule of thumb" 
suggested by Tallmadge (1977, p. 34). Effect size is calculated by subtracting the 
posttest mean scores of experimental and control groups and dividing the remainder 
by the standard deviation of the control group, (see Glass, 1978). An effect size 
can range from 0% to 100% (or 0 to 1 if converted to fractions). For the CKT, an^ . 
effect size of 25-50% was the criterion used to determine practical differences.. 
For the OPIP, an effect size of 10-25% was used. A lower effect size was expected 
from the OPIP because attitudes are difficult to' change in any short-term treatment 
(Shaw and Wright, 1967; Hepburn and Napier, 1980b). 



\ 



Table 7 



Descriptive Statistic^ for Main Effects of Group, Gender, and Race in Secondary aird^Elementaxry 
Evaluation Samples on Knowledge and Attitude Pretests and Posttests 



Source 



Main 
Effect 



Class 



N 



Knowledge (CKT) 

Posttest 
Pretest Unadjusted Adjusted 
Mean S . D . Mean S . D . Mean 



AttitjtideTOPIP) 



Pretest 
Mean S.D« 



Posttest 
4toad j us ted Ad j us ted 
Mean S.D. Mean 



: . Group 


Pro j ect 
Control 


165 
180 


31.9 
29.7 


12.5 
10.3 


.36.8 
30.3 


12. 
10. 


5 
9 


35.8 


122.0 
119.3" 


9.3 
9.9 


r- — - -■ 

123.2 
119 . 2- 


.9.9 
10.5 


122.0 

• 

120.4 


Secondary Gender 


Mai o 

Female 


1 57 
X.JI 

18d 


JU . J 

31.2 


11.5 


^9 7 

34.0 ; 


1 9 

Li. . 
11. 


j 
7 


33.4 ^ 


1 1 A Q 

LLO . 7 

122.1 


i n i 

LVJ i L 

9.1 


11875^11.2 
123.3 .9?2 


1 90 ft 

-a?.2.i 


Race 


White 


■208 


33.7 


11.5 


36. 6 J 


u. 


9 


34.1 


' 122.3 


9.2 


122*. 7 


9.9 X 


\i2ir?^ 


Black 


137 


26.3 


9.8 


28.5 


10. 


8 


32.3 


118.1 


%.9 \ 


118.7 

* • 


10.8 


1^0t6 


Group 


Project 


110 


24.2 


7.0 


26.5 


7. 


7 


irk 


110.8 


12.9 


113.2 


14.4 


114.1 


Control 


104 ■ 


26.4 


6.5 


26.1 


7. 


5 


26.0 


112.7. 


12.0 


113.7 


13^8 


1 112.6 


Elementary Gencf&r 


Male 
Female 


Ill 
103 


25 a 
25.4 


7 .'2 
6.4 


26.5 
26.9,. 


. 7. 
7. 


./ 

5^ 


26; 7 T 
26.8 


112:0 
111. 4 


11.9 
13.1 


112.8 
114.1 


14.6 
13.6 


112.6 ' 
114.51 . 


Race 


-White 


117 


27.6 


5.9 


28.2 


6. 


9 " 


26 . 3 


U4.0 


12.2 


IKS 6 


12.4 


114,5 


Black 


97 


22.5 


6.9 


24.9 


8. 


0 


27.2 \ 


109.0 


12.3 • 

s 


159.6 


15.0 


112,0 
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£0 



t 

. \\ ■ 


• 


0 

\ 


* 
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Table 8 






«■ 




• 




* « 

Analysis 


of Covariance 


Tests on Knowledge and Attitude Posttest 




V 




». 

M 

• \ 


for Secondary and Elementary Evaluation Subjects" " 






• 






»• 


Secondary 






Elementary 




Source 




Knowledge 


(CKT) - Attitude 


(OPIP) 




Knowledge 


(CKT) 


Attitude 


(OPIP) 


* 


or 


MS 


F MS 


r 




MR 


r 




F 


. Covariate 




33233.3 


756.8* 20663.2 


439.4* 


* 1 


6529.1 


248.2* 


28517.4 


448 . 9* 


Group . 


1- 


1776.2 


40.5* ' 205.7 


'4,4 


1 


; ,93.9 


3.6 


114.4 


1.8 


. Gender 


1 


0.1 


• 0.0 376.8 


* 

8.0 




• 0.6 


* j 
010 


131.7 


2.1 — — 


Race 




,\ 219. 78 


5.0* 55.2 


1.2 




31 \ 


1.2 


319.3 


5.0* 


Group x Gender 


• . 1 


' 24.46 


0.6. 10.6 


0.2 . 




- 15.4 


0.6 

* 


* 0.7 


0.0 ' ' 


X- Group x Race 




30.6" 


0.7 : 25.4 


0.5 




159.2 


6.1*: 


0.3 


0.0 . - 


Gender x Race 




151.0 


3 .,4 .0.0 


0.0 




12.2 


0.5 


36.0 


- 0.6 


Group x Gender x Race 

• 




127.6 


2.9 * . 42.3 


0.9 




26.3 


0.7 


3.1 


0.1 


Residual 


336 


43.9^. 


47.0 




205 

£m \J J 


26.3 




63.5, 




Total 


344 


146.5 


108.3 




213 


57.6 




198.0' 




* 

Significant at the £ ^ 

* 


.05 level 


* „ 
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<- 
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1 

22 





I The effect size found for 'the CKT was 422, and for the OPIP 11%. Thus, the / 
answers to product evaluatioaquestions were positive for the secondary evaluation. 
The ICE Project did' significantly and practically affect the political knowledge and 
attitudes of secondary students. \ i 

\ Elementary product results . Table 7 also presents the descriptive statistics 
for the elementary students on the CKT and O^IP tests. Analysis of covariance 
results on the CKT and OPIP posttest comparisons between project and control students 
(Table 8) indicated no significant difference between project and control groups 
on either test. Therefore7~an^^ were 
negative for the elementary .project evaluation. The political knowledge and attitudes 
of elementary students were not significantly effected. 

There were circumstances related to the implementation of the ICE Project which 
could have accounted for the negative findings in the elementary" eval uat ion. 
Originally, a ten week implementation <had been designed for boffi" secondary and 
elementary programs. For elementary classes the average daily time devoted to content 
was 23.8 minutes, while the secondary average was 44.4 minutes. Obviously four* 
additional weeks ef treatment would be more essential for elementary students. 
However, report deadlines set by the fynding agency had forced the shorter six-week_- 
treatment. Cook, Cook,- and Mark (1977) warn that failure to' give 'sufficient 
exposure to- process results in loss of statistical power. Also, because two pro- 
ject teachers did pot retum-the three questionnaires documenting process, the 
evaluators wondered whether they had actual ljT implemented the prbject properly. 
Staff and reviewers decided that re-evaluatipn in -elementary grades Was necessary 
before final concl usions. about the effectiveness of the project at this level. 

Supplemental information on gender and race . Although product evaluation 
questions did not- call for examination of gender and race, data on these two 
variables were col lected.and used in the statistical analysis to send some light on 
the effects on student sub group's. There was a significant di fference between 
races on the knowledge test (CKT) and between genders on the attitude test (OPIP). 
Hqwever, no significant interactions were found which means that gender ^did not make 
a significant difference in performance by project or control and race did not make 
a difference by project and control. Nevertheless, within project and control groups 
whites scored better than blacks on knowledge, and females scored better than males 
on attitudes (see Table 7) 
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Procedures for Re-evaluation 

Subjects . . 

Teachers . In the re-evaluation study 20 elementary teachers were originally 
involved. (These were^different teachers than in the first evaluation.) These 20 
.project teachers werematched with 20 control teachers in the same^manner used in 
the initial evaluation. Unfortunately, one control' teacher left the school system 
and the matched project teacher? was dropped from the study. Also, twb project . 
teachers never returned daily \ogs and they with their control matches were dropped. 
The final sample of teachers consisted of 17 project and 17 control teacWs all 
matched by grade (T first; 2 second; 4 third; 2 fourth; 2 fifth; 3 sixth; and 3 

seventh). i 

i 

Students . TheUtudents involved in the re-evaluation were in classes taught by 
project^ and control ^teachers. Initially, '611 students (290 project; 321 control) took 
the pretests. However, theyfinal sample size was 546 (260 project; 286 control) — 
approximately a 10% i loss for each group. The loss reflected the fact that only students 
who took both pretests and posttests were included in the final sample. The final 
sample had 272 males (132 project; 140 control) and 274 female% (128 project; 146 
control). Racial composition .was 264 white (129 project; 135 control) and 2&2 black 
(131 project; 151 control). *f 

Measurement Instruments \ 
Process instruments . Only the three questionnaires-Questionnaire on ICE 
objectives, Questionnaire on Management Support, and Questionnaire on Materials were 
used in the re-eya]uation. These instruments were identical to those described 
earlier in the Procedures for Evaluation section. ' 

c 

Product instruments . Again the elementary forms of the CKT and 0PIP were used 
to collect data from the student subjects. Cronbach alpha reliability estimates of 
the .iota 1 CKT in the re-evaluation was .85 for the pretest, and .84 for the post- 
test. Cronbach alpha reliability estimates for the total OP IP score were .83 for the 
pretest, and .83 for the posttest. 

Treatment ♦ / 

Teachers . The project teachers, none of whom were previously involved with the 
project, received staff development training similar to that in the initial evalua- ° • 
tion. The' major difference was in the length of the 'treatment. In the re-evaluation, 
five days tff staff development (approximately 40 hours) was used, but the major 
elements of the/original staff development treatment were maintained, and the staff 
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jfelt the treatment was equivalent. Since the effectiveness of the staff develop- 
ment process was documented in this evaluation and'in an earlier study (Hepburn 

• ' . ■ * 1 

and Napier, 1979), it was not monitored again. 

Administrators, Again meetings' were held with principals in project schools, 
to make them aware of objectives and methods of the project. They were encouraged 
to be supportive to teachers involved in the project. . 

• Students. Project teachers followed* the objectives", and utilized the materials, 
includ.ing x a. finalized version of the elementary handbook, and with administrative 
support taught their citizenship units over a ten-week period* Again, each project 
teacher was Vree to operational 1ze "the units taught* Therefore, daily logs were 
again kept by ^oth project and control teachers to document daily treatment to 
students* Analysis of the log data Indicated, that the project teachers spent s1gni- w 
flcantly rare time covering contents related to the objectives; and taught using 
signif1cantly\more x of the topics, materials, and activities than control teachers 
(see. Table 9) A ; \ 

Data Collection \ 

Process evaluation data* All 17 of the el emerfCary pro ject teachers responded 
to the three questionnaires on project objectives, management support, and materials 
at the end of the ten week implementation in the Fall, 1980* 

Product evaluation data * Project and control elementary students took the two 
pretests at the beginning of Fall quarter 1980* ^he attitude test (OPIP) was given 
orie day, and the knowledge test (CKT) was given the^ext day* '^Posttests were given 
in November at the end of "the ten weeks of treatment," and the tests were given 1n 
the same order. Since 1t was impossible to arrange to retrieve missing data, only 
those students who took all four tests were included in'the evaluation. 

Re-evaluation Results 

Process Evaluation 

Table 1CT presents descriptive data for responses to the first part of each 
questionnaire- Results again indicated that objectives and materials were per- 
ceived as very ^helpful. Unlike the initial evaluation, the project teachers per- 
ceived management support in the re-evaluation as very supportive* 

Product Evaluation . 

Again, analysis of covariance procedures were used blocking on gender and race- 

» 

Descriptive statistics for student tests are in Table 11. The analysis of covariance 



• V ' s Table 9 

descriptive Statistics, F-Ratios, P-yalues for Number 
** of Materials, Activities, and Contents plus- Time 

m — * |* 1 ' t 

Covering Topics used by Elementary Re- evaluation 
Pro j ect and Control - Teachers 



Source 


'Statistic Project, 


* Control 


F 


P 






• (16*) 







Number of 


"Mean 


6.0« . 


3.9 


•Materials . 


S.D. 


1.4 


1.5 


(Scale 1-8). 




* 


Number of • 


Mean 


5.8 


3.4 


Activities ( 


S-.D. 


-0.8 


,1.3 


(Scale 1-8) 






* 


Number of 


Mean 


7.4 


» -3.8 


Contents 


S.D, 


1.4 


2.3 


(Scale 1-8) 








Time ' on 


Mean 


1114. 4 


285.0 


Topics 


S.D. 


. 632/5 


307.2 



17.55 



37.41 



000 



»000£ 



29.57 .000 



22.49 ..000 



*. 



One Control Teacher never returned the daily log 



! * 



L 
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Table 10 

"Descriptive Statistics for Responses to Questionnaire 
• on ICE Objectives T Management Support and Materials 
., 'for Re-evaluation Project Teachers 



* 






Source 


Statistic 


Pro j ect 


♦ 


Mean 


* 3.5 


Objectives 


* - 

S.D. 


0.5 


(Scale 1-4) 


N' 


17 


it. 

Management 


Mean - 


3.8" 


Support 


S.D. , 




(Scale 1-4) ' 


* ' y N 


17 

* 




Mean 


-3.8 


'Materials 


1 

■ ' S.D, 


0.6 


(Scale 1-4) 

a 




* \ 1? . 



^ * 



Table 11 



Descriptive Statistics for Main. Effects of Group, Gender, and Race in Elementary 
Re-evaluation Sample on Knowledge and Attitude Pretests and Posttests 



Source Class 



N 



Knowledge (CKT) 

Post test 
Pre tes t Unadj us ted Adj us ted 
Mean S . D . Mean S . D . Mean 



Attitude (OPIP) , > 
Pcrsttest 

Pretest Unadjusted Adjusted 
Mean S.D. Mean S.D. Mean 



Group 



Project 260 23.8 
Control 286 23.2 



6.8 26.7 
7.1 24.3 



6.3 
7.3 



26.4 
24.5 



108 . 6 11. 7—1-1-3.2 12.2 H3t6- 

109.8. 12.1 112.2 12.8 111.7 



Gender 



Male 



272 23.5 



Female 274 23.5 



7.2 25.3 7.1 25.3 109.0 12.3 112.0 12.9 U2.0 
6.8 25.6 6.7 25.6 109.4 11.6 113.3' 12.0 113.2 



Race 



White 264 25.3 6.6 27.0 6.7 25.7 

t 

Black 282 21.8 7.0 24.0 6.8 25.2 



110.7 11.0^ 114.7 12.'5 113.6 

107.8 11.7 110.7 12.2 111.7 



TTfrteJZ), indicated, a significant 'difference, between project and control groups 
on both thTlttitucte aTTdHmcftiledge €est^/ The effect .size for the CKT was 26%, and 
for the OPIP 12% in dicating positive- answers to both product evaluation, questions . 
Bafe*d^on\ re-evaluation results, -the ICE Project model does significantly and prac- 
tically affect-the political knowledge and attitudes of elementary students. 
SupplementaT~I¥fonnatitMuon tiendeF~ancntgc~e " r " 

Examination of Table 1 indicated that there were no significant differences- 
between gender on either the knowledge (CKT) or altitude (OPIP) tests>-Also, the 
ms no significant difference between races on knowledge; but there was a signi 
cant- difference, between races on attitude with whites scoring' higher (Table Tl 
However, again there were no significant interactions which means race did not 
_make.JU£ i.gn i f i cant differenc e in performance by projec t and cont HT^roup^, 
project and control gr^ps, 1 whites did -score better than bracks orT^ttitude.' 




-Conclusions, 



-il^The^IlnprDving Citizenship Education Project-Was evaluated in the real world of 
; the, school system where many factors human and mechanistic prevent the, implementa- ' 
(tion of an ideal evaluation design. Holiday vacations, student absences, change 
11 n class schedules, teacher job changes, and the failure to return a few question- 
naires — these are some of the many problems that quite naturally 'compromise any 
atStempt to. conducts i3eal evaluation. In another paper, the authors have dis- 



cussed criteriefof useful project evaluation design, which though modeled on eval- 
uation ideals, must realistically Be modified to meet 'limitations of the fielcf. 
setting in which it is applied (Napier and Hepburn, 1981). In this section we will 
focusjan^specific limitations of the actual design used and review the results for 
t^i>HEfrti cations studies,curriculum change. 

Limitations ' y' 

I It must first be emphasized that results of the process evaluation in this 
~stu4y apply only to "process" .as operational ized by, four process instruments (staff 
development test and three questionnaires) and^the daily teacher log. Likewise, 
results of the product evaluation apply only to products which are operational ized 



by jthe specific objectives of^tRe^prqject and the. knowledge and attitude instru- 
ments which assessed the student outcomes. 

i Second, the internal validity' of the study was not absolutely assured, although 
teachers and students were matched and qu^Si -experimental design was used in the 
product evaluation. Orje source of rival causes was not controlled with the evalua- 



tion desian 



used— the interaction of $election and maturation or history. Also, 
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Table 12 



ysls of Covartajice Tests on Knowledge and Attitude Posttest 
for Elementary Re-evaluation Subjects 



Source 



Knowledge (CKT 



Attitude (OPIP) 



Co variance" 




* 1/iO/i7 Q 


* 




f%f\l ft 

DO / • 1/ 


\ 

Group 




482.3 


* 

25. 1*_ 


503.1 


7.2* 


Gender 




9.6 


0.5 


108.0 


2.7 


Race 




26.3 


,1.4 


439.5 


* 

6.3 


Group x Gender 




29.6 


• 1.5 


11.1 


0.2 


Group x Race 




7.9 


0.4 


42.1 


0.6 

o 


, Gender x Race 




5.6 .. 


0.3 


2.3 


0.0 


Group x Gender x ftace 




70* 8 


3.7 /: ; 

y 


54U ' 


0.8 


Residual 


537 


19.2 


C ' 


69.6 




Total . * 


545 


. 4.7.5 




155.9 





Significant at the j> ^ .05 level 



. since pre-experimental design was used to [examine the process, the effectiveness of 
. the process could be the result of such rival causes as past history, selection, 
Hawthorne effect, and scyforth. Thus, Results of the study could be attributed to 
extraneous variables. * 

A third limitation is in the statistical validity of the evaluation and re-, 
evaluation. Analysis of covariance with blocking was the statistical procedure 
used to analyze product data. Possibly different analysis procedures would produce 
different results. 

Finally, the external validity of the study may be questioned since volunteer 
teachers weije used and only students who completed all pretest and posttests were 
analyzed* It 'seems likely that non-vdlunteer teachers and non-attending students 
would produce different results. Also, only one school system was used in the 
evaluation. Again, it, seems Jlikely- that teachers and students in different school 
systems could produce-different^results* " r 

« On revie^ of the above-meantioned limitations, the evaluators regard the 1s- 

* } 

"sue of general i zability of the instrumentation as a minor one. The instruments 
used \ Vfc the evaluation of student knowledge and attitude outcomes were general, . 
norm-r y eferenced measure which had extensive piloting and analysis in validity 
studies. 'Moreover, most school systems find it feasible to evaluate educational 
programs -with such paper and pencil measures. '* 

-^-Regarding internal validity, in defense of the. design, consideration should 
; be given to the fact that teachers and student groups were matched carefully. 
Also, the use of analysis of covariance procedures with blocking lessened the 
likelihood of initial differences as an internal validity problem.- 

The- use of analysis of covariance with blocking was not viewed as a major 
limitation either. Simple inspection of the descriptive statistics indicated 
that** differences between groups, genders, or races were logical. 

Only external validity was considered to be a serious limitation. The citi- 
zenship education model may not work with other teachers, students, or school 
systems. Consequently, replication with other teachers and students in other 
school settings is the appropriate way to verify the conclusions of this evaluation. 
Implications 

Results'of the test of the ICE model suggest implications for social studies 
* cumculum change which are significant. The model centered on teacher involve- 
ment and teacher behavior as a key factor in generating improved student political 
knowledge and attitudes. If the results of this evaluation are valid, then it is 

\ , 

00 



, * ! ... 

logical that any attempt to modify social studies curriculum and instruction must 
(involve teachers. Likewise, the involvement of principal'S~Vea'ders- w1th""the~T6cal" 
school system, and community leaders appears important to the effectiveness of 
improvement projects. Without the combined support of these actors, change in the 
curriculum might not occur. 

The findings indicate that clear explication of objectives _and an explanation 
of how-tfie objectives are linked to content and teaching methods contribute to the' 
success of a curriculum change project. Likewise, teacher education which provides 
content information and instructional ideas and materials related to the objectives 
-also plays a role in generating change .in the classroom. These components provide 
teachers with the intellectual framework and the classroom tools to integrate and 
adapt_th.e_mo.dif i ed.curri cul um„at their grade leve l . Probably a major impetus for 
.change came from the attitud'inal effects- of the cameraderie of a group of teachers 
working together in a°staff development program where thefr -involvement is high- 
lighted. All support components were aimed in their direction. ' They developed 
their own study units- f or students, and they were considered the important actors 
in the project. The result was that they changed what they were doing in their- ' 
classrooms in regard to citizenship education, and the ultimate outcome was that 
student performance on 'knowledge and attitude tests, regardless of gender or race, 
showed significant positive changes. 

Again, it should be. made clear that the Improving Citizenship Education Pro- 
ject was designed to enhance citizenship e'ducation as defined by the specific ob- 
jectives of this project->-which are essentially political knowledge, political 
skills, and political attitude objectives. Given a different set of objectives 
this multiple support model centered on the teachers might work as^well. However, 
we can only claim that the model is valid and effective for the .ICE project. . 

' Currently, the ICE Project model is being disseminated throughout the State 
of Georgia with approximately 20 school systems and 67 schools in the process of 
adopting the change model and-utilizing the guiding materials. Students in these 
adopting schools are being tested, and from the dissemination information, the 
replicability of 'this evaluation will be discovered. 

* i 
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I Information about the, final versions of the Secondary and Elementary Handbooks 
can be obtained from: , 

' Sheila L. Margolis , \ t 

Improving Citizenship Education Project .if 
Liberty-Gunn Center 
4820 Long Island Drive 
Atlanta, Georgia 30342 

While there are no established criteria indicating that a given effect size is 
of practical significance, the work of Cohen* (1977) represents the most useful 
Qurce J _Basically, Cohen defends using the idea of small, medium, and large effects. 
A small effect (approximately .ft^is a change which is. unlikely to be observable 
in outward behavior, .but which h|s theoretical value. A medium effect (approximately 
.25) is a change whichrhas the possibility of perception in outward behavior and the 
effect most often found in psychological research.. Finally, a large effect 
(approximately .50) is a change-which 4s seen in outward behavior,- but is a rare 
finding .in single psychological research studies. In this study a small effect 
was .expected for attitude changes while a medium effect was expected for knowledge 
change. Attitudes are difficult to alter, yet any positive change would be of value 
to a curriculum project. Knowledge is more easily changed, but since the knowledge 
test was a general norm-referenced instrument, a large effect was' unlikely/ 
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